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Abstract 


the  number  of  machines  in  a  computer  installation  increases,  the  likelihood  that  they  are 
being  equally  used  is  very  small.  We  have  implemented  a  load-balancing  system  to 
increase  the  overall  utilization  and  throughput  of  a  network  of  computers.  With  this 
system,  a  busy  machine  will  locate  an  underutilized  one  and  attempt  to  process  certain  types 
of  CPU  intensive  jobs  there.  We  present  here  a  complete  functional  description  of  the 
system  and  an  analysis  of  its  performance. 
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1.  Introduction 

Id  any  multi-machine  computing  installation,  there  is  a  need  to  evenly  distribute  the 
workload  over  the  participating  machines.  Otherwise,  some  machines  will  remain  idle  while  others 
become  overloaded.  The  Berkeley  UNIX  environment,  with  its  networking  capabilities,  provides 
for  the  interconnection  of  powerful  processors.  The  busier  machines  may  move  tasks  to  the  less 
busy  machines,  offering  a  more  even  distribution  of  workload  across  the  entire  system,  and  a 
decrease  in  overall  command  response  time. 

This  paper  describes  one  load-balancing  system  currently  in  use  in  the  EECS  Department  of 
the  University  of  California  at  Berkeley  called  Maitre  d*.  For  a  given  class  of  relatively 
expensive  jobs,  Maitre  d'  will  attempt  to  locate  a  lightly  loaded  machine  and  process  the  job  at 
that  machine.  In  this  way,  imbalances  in  processor  demand  across  machines  can  be  smoothed 
through  an  automatic  redistribution  of  the  load. 

This  paper  is  broken  into  several  sections:  background,  functional  description,  operational 
considerations,  implementation  notes  and  performance  analysis. 


S.  Background 

Maitre  d'  was  originally  proposed  by  several  students  at  Berkeley  to  relieve  peak  usage 
demands  on  the  machines  used  by  the  Computer  Science  Division  of  EECS,  particularly  those 
dedicated  to  instruction.  These  machines  had  always  been  VAX  ll/750s  and  ll/780s, 
characterized  by  extremely  uneven  workloads.  Research  and  administrative  machines  alike 
suffered  from  high  daytime  loads,  while  being  relatively  idle  at  night.  Instructional  machines 
would  go  unused  for  long  periods  of  time,  but  would  become  so  loaded  in  the  days  prior  to  an 
assignment  being  due  that  they  would  become  almost  unusable. 

In  December  1084  the  university  received  a  gift  of  six  VAX  11/750's  from  Digital  Equipment 
Corporation^]  as  part  of  a  grant  earmarked  for  undergraduate  research  and  instruction.  These 
machines  were  not  ready  for  assignment  to  formal  classes  when  the  school  semester  began,  but 
they  were  accessible  through  a  10  megabit  etbernet  [1,6]  connection  with  the  rest  of  campus. 
Consequently,  the  new  VAXes  were  designated  as  remote  process  servers  for  overloaded 
instructional  computers  being  rented  by  the  Department,  with  Maitre  d"  acting  as  the  agent. 

S.l.  Functional  Description 

Maitre  d'  operates  around  a  modified  state-broadcast  algorithm.  Every  potential  client 
maintains  a  list  of  known  server  machines.  Associated  with  each  server  is  a  binary  value 
representing  that  server's  availability,  as  determined  and  advertised  by  the  server.  When  a  user 
invokes  an  application  modified  to  run  under  Maitre  d\  a  decision  is  made  as  to  whether  the  job 
should  be  performed  remotely  r  r  on  the  user's  machine  by  comparing  the  UNIX  five  minute  uad 
average  against  a  minimum  load  threshold.  3  If  it  is  determined  that  a  remote  machine  should  be 
used  (local  load  average  >  sendoff  threshold),  the  list  of  known  servers  is  consulted,  and  the  least 
recently  used  available  server  in  the  list  is  chosen  to  perform  the  job.  That  prospective  server  is 
contacted,  informed  of  its  selection  and  then  told  to  perform  the  service. 

The  process  of  selection  and  contact  can  be  made  entirely  transparent  to  the  user.  This  is 
especially  necessary  in  an  instructional  environment  where  most  users  are  naive  and  unable  to 
handle  exceptional  conditions  (such  as  failure).  They  care  only  that  their  job  gets  done,  and  not 
where. 


1  Mtitrt  4'  comes  from  the  French  term  mtitrt  iXold,  meaning  hotel  or  dining  room  men  age  r. 

*  The  UNIX  five  minute  toad  average  ie  defined  ai  the  average  number  of  jobt  in  the  run  queue  exponer  Unity 
smoothed  over  the  put  five  minute*.  It  it  a  limited  metric,  but  very  cheap  to  obtain.  For  a  more  complete  evaluaticn  of 
bad  metrics,  see  |2]. 
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Maitre  d' can  be  used  to  off-load  almost  any  type  of  Don-interactive  job.  The  initial  version 
of  Maitre  d'  exported  pascal  and  C  compiles.  It  has  since  been  expanded  to  include  typesetting 
(troff),  circuit  simulation  (spice),  a  true  Pascal  compiler  (pc),  and  others  (see  details  in  Appendix 
B).  New  applications  are  being  added  as  need  and  opportunity  arise. 

3.  How  It  Works 

This  section  describes  the  operational  details  of  Maitre  d\  Readers  uninterested  in  the 
mechanics  of  the  system  should  skip  to  section  four. 


3.1.  Establishing  Connections 

Before  considering  the  total  operation  of  Maitre  d',  it  is  important  to  review  some  of  t..e 
basics  of  interprocess  communication.  This  section  describes  the  establishment  of  connections  and 
the  relationship  between  the  client  and  server  machines.  It  assumes  no  familiarity  with  UNIX 
Interprocess  Communications  (IPC).  We  present  a  brief  review  of  IPC  under  UNIX  in  the 
following  paragraph.  The  reader  unfamiliar  with  UNIX  IPC  may  wish  to  consult  either  [5]  or  [7] 
for  a  more  thorough  description. 

Separate  processes  may  communicate  with  one  another  using  any  of  several  alternative 
methods.  We  describe  here  only  the  user-level  protocol  for  a  connected,  bi-directional  stream 
communication  capable  of  crossing  machine  boundaries.  The  terms  client  and  server  are  used  to 
describe  the  parties  during  the  connection  phase.  The  server  is  generally  referred  to  as  a  daemon, 
or  a  process  which  runs  indefinitely,  usually  blocked  while  waiting  for  an  event.  The  server 
daemon  listens  at  some  well-known  address  *  waiting  for  requests  for  service.  The  client  calls  the 
server  via  some  high-speed  communications  medium  (in  this  case,  the  etbernet)  and  requests  a 
connection.  Implicit  in  this  client's  call  is  the  requester's  originating  address.  The  server  accepts 
by  completing  the  connection  with  the  client.  Once  the  connection  has  been  established,  general 
practice  is  to  have  the  server  create  a  duplicate  server  process  which  interacts  only  with  the 
initiating  client,  leaving  the  original  server  free  to  continue  listening  fo.  further  requests.  Once  the 
connection  is  established,  both  the  client  and  server  may  read  from  and  write  to  their  common 
connection  as  though  it  were  a  standard  UNIX  file.  This  makes  data  transfer  between  the  two 
processes  extremely  simple. 


3.2.  System  Components 

Load  balancing  under  Maitre  d’ comprises  four  distinct  components: 

(1)  maitrd  * 

All  machines  which  are  to  be  able  to  off-load  jobs  to  other  processors  run  a  maitrd  daemon. 
This  process  maintains  the  list  of  all  server  machines,  including  status  information  as  to 
whether  or  not  they  are  currently  willing  to  accept  jobs.  For  the  remainder  of  this  paper, 
the  terms  maitrd  and  client  may  be  used  interchangeably. 

(2)  garcon 

This  is  the  server  daemon  running  on  each  machine  which  is  to  be  a  compute  server.  It  has 
two  functions:  maiLt^ining  status  connections  with  client  machines,  and  accepting  jobs  from 
application  programs.  Garcon  and  server  may  be  osed  interchangeably. 

a  An  addre**  is  the  combination  of  the  machine’*  internet  addree*  and  a  local  port  number. 

4  M litre  d’  (with  a  capital  M  and  spelled  correctly)  is  the  name  of  the  system.  One  of  the  component  protrams  is 
called  maitrd  (with  a  lower  case  m  and  modified  spelling). 
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(3)  application  program s 

The  maitrd  and  garcon  component*  provide  only  n  control  environment  through  which  the 
execution  of  programs  may  actually  be  distributed  among  many  processors.  The  software 
that  provides  the  interface  between  the  user's  requested  task  and  Maitre  d'  is  referred  to  as 
the  application. 

(4)  miscellaneous 

This  includes  the  black-box  library  routines  used  to  interface  with  maitrd  (described  later), 
and  a  dynamic  control  program  called  mdc  used  to  tune  parameters  while  the  system  is 
running  (appendix  C). 

When  a  maitrd  process  is  started,  it  tries  to  create  control  channels  with  the  garcon 
daemons  running  at  a  pre-design ated  set  of  servers.  This  set  is  given  in  a  startup  configuration 
file  (appendix  B)  kept  at  the  maitnfs  machine.  Through  these  control  channels,  the  maitrd 
process  can  determine  which  machines  are  up  and  accepting  jobs,  which  are  up  and  not  accepting, 
and  which  are  down. 

The  counterpart  to  a  maitrd  process  is  the  garcon.  This  daemon  listens  at  its  well-knowc 
control  port  for  requests.  When  a  connection  from  a  maitrd  is  accepted,  the  garcon  daemon  lets 
the  maitrd  know  whether  the  garcon  host  is  will:ng  to  accept  jobs.  A  server  declares  itself  read; 
if  its  UNIX  five-minute  load  average  is  less  than  some  threshold  and  there  are  fewer  than  a  given 
number  of  active  users  logged  in  to  the  server  machine.  Both  of  these  thresholds  can  be  set  from 
a  configuration  file.  After  the  connection  is  made,  garcon  checks  its  own  status  every  30  seconds, 
informing  each  of  its  connected  clients  (i.e.,  multicasting)  whenever  availability  changes. 

The  typical  configuration  for  Maitre  d' is  to  have  a  set  of  machines  in  which  each  runs  both 
the  maitrd  and  garcon  daemons.  Each  machine  in  the  cooperative  would  look  tor  help  from 
others  whenever  it  became  too  busy,  and  in  return  would  be  willing  to  take  on  jobs  from  its  peers 
when  it  would  otherwise  be  idle.  One  alternative  to  this  is  to  introduce  dedicated  servers  into  the 
system  (machines  running  only  garcon ),  which  accept  jobs,  but  never  send  them  out.  A  second 
alternative  involves  running  a  clearinghouse  maitrd  and  is  discussed  below. 

For  each  client/server  pair,  there  is  an  open  stream  connection  maintained  for  the  duration 
of  the  relationship.  This  imposes  a  limit  on  the  number  of  servers  that  may  be  kept  track  of  by  a 
single  maitrd  process*  Although  not  particularly  cumbersome  for  a  small  number  of  machines, 
the  total  number  of  status  connections  for  N  clients  and  M  servers  grows  as  NM,  and  is  order 
N 5  when  all  N  machines  are  accommodating  both  garcon  and  maitrd.  To  slow  this  growth, 
Maitre  'd  has  the  capability  for  clearinghouse  clients.  Instead  of  running  a  maitrd  on  every 
machine  that  is  to  offload,  applications  can  request  an  available  machine  from  a  remote  maitrd. 
This  is  most  effective  in  clusters  of  diskless  workstations.  It  is  sufficient  to  have  a  single  maitrd 
running  on  the  file  server  handling  requests  from  all  of  the  file  server’s  clients.  Any  reliability 
gained  from  having  redundant  maitrd’s  would  be  pointless,  as  the  workstations  aren't  very  useful 
when  the  file  sen  r  is  d^wn. 

Certainty  of  state  is  the  main  reason  why  open  collections  are  used  for  the  control 
channels  A  status  update  from  a  garcon  is  guaranteed  to  arrive  at  each  connected  maitrd.  If  an 
update  is  undeliverable,  garcon  assumes  the  intended  maitrd  no  longer  exists  and  removes  it  from 
its  multicast  list.  Similarly,  if  a  garcon  disappears,  UNIX  IPC  enables  each  connected  maitrd 
daemon  to  recognise  this  immediately  and.  mark  tNe  associated  server  machine  as  unavailable. 
The  maitrd  daemon  attempts  to  re-establish  connections  to  downed  servers  every  five  minutes. 

S.S.  Selecting  A  Machine 

The  maitrd/garcon  connection  performs  no  real  work.  Its  only  purpose  is  to  give  the  local 
maitrd  a  pool  of  processors  from  which  it  may  choose  a  server.  In  addition  to  maintaining 


*  la  4.2BSD,  this  limit  ni  Kt  by  the  operating  lyitem  at  20.  The  newer  4.JBSD  allow*  04. 
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connections  with  remote  servers,  maitrd  nlso  listens  in  on  »  second,  local  socket  for  requests  from 
an  application  program,  which  is  any  program  that  has  been  modified  to  run  under  Maitre  d'. 
These  applications  first  connect  to  the  local  maitrd  process,  asking  for  an  available  machine.  If 
the  local  load  is  less  than  the  sendoff  threshold,  or  no  remote  servers  are  presently  available,  the 
application  is  told  to  perform  the  job  locally.  Otherwise,  maitrd  does  a  round-robin  traversal  of 
its  list  until  it  finds  a  machine  where  the  gareon  has  advertised  a  willingness  to  accept  jobs.  It 
passes  back  to  the  application  program  the  internet  address  of  this  server  and  terminates  the  local 
connection  with  the  application. 

8.4.  Program  Execution 

In  addition  to  listening  on  its  control  port,  gareon  also  listens  on  a  data  or  service  port.  It  is 
this  address  that  maitrd  gives  to  the  application,  and  it  is  the  responsibility  of  the  application  to 
create  the  connection.  Once  the  connection  is  created,  the  gareon  process  creates  a  copy  of  itself. 
This  copy  communicates  with  the  local  application  and  executes  the  requested  task  on  the  remote 
machine,  leaving  the  original  gareon  free  to  handle  further  requests  from  other  applications.  All 
communication  between  the  application  and  the  server  is  done  through  this  data  port. 
Commands  and  data  are  passed  from  the  application  process  to  the  remote  machine;  results  and 
an  exit  status  are  passed  back  from  the  server  to  the  application  process. 

Figure  I  shows  a  typical  Maitre  d'  interaction  occurring  in  three  stages.  Permanent 
communication  streams  are  shown  in  thick  black;  temporary  ones  in  thin.  In  stage  I  the  user 
invokes  some  application  (compiler,  formatter,  etc.)  which  connects  to  the  local  maitrd  and 
requests  an  available  server.  This  maitrd  has  been  maintaining  status  connections  with  many 
gareona  (only  one  shown  in  the  picture).  The  client  daemon  passes  a  server  address  out  of  its 
maitrd  port  (A)  to  the  application  program  and  terminates  its  connection  with  the  application. 
Stage  II  shows  the  application  connecting  to  the  gareon  port  (C)  that  was  returned  by  the  local 
maitrd  and  requesting  a  service.  If  the  application  and  request  are  valid  (see  sec.  4.1.2),  the 
server  creates  a  copy  of  itself  (called  forking  in  UNIX).  Note  that  the  service  port  is  passed  down 
to  the  newly  created  dedicated  child  process  (C*).  The  dashed  lines  leading  into  the  gareon  port 
indicate  that  the  parent  stops  attending  to  the  application  on  the  other  end  of  the  channel  once 
the  child  gareon  has  taken  over.  Stage  III  has  the  dedicated  gareon  creating  the  process  to 
perform  the  requested  task  and  acting  as  a  data  buffer  between  the  application  program  and  that 
task.  When  the  requested  process  finishes,  its  exit  status  is  returned  to  the  application  at  the 
originating  machine,  and  the  child  gareon  terminates. 

8.5.  I/O  Handling 

The  C-shell  provides  every  process  with  three  default  file  deaeriptora  or  data  channels: 
standard  input  (atdin),  standard  output  (atdout)  and  standard  error  (atderr).  Stdovt  and  at  den¬ 
ote  both  output  channels;  atdin  is  the  only  input  channel.  Many  UNIX  programs  are  capable  of 
taking  their  input  from  atdin,  and  placing  their  output  on  atdout.  Convention  has  error  messages 
going  to  atderr.  All  processes  return  an  8  bit  status  value  upon  termination. 

Stdin  to  a  gareon  task  is  passed  directly  from  the  originating  machine.  But,  on  the  return 
path  of  the  service  channel,  Maitre  d’  provides  only  one  data  stream.  The  extra  bandwidth 
needed  to  handle  atdout,  atderr  and  the  exit  status  is  obtained  by  returning  all  data  from  the 
server  in  finite  packets,  and  prepending  each  packet  with  a  four  byte  header.  The  first  byte 
indicates  the  type  of  data  being  returned:  atdout,  atderr  or  exit  status.  If  the  header  indicates  an 
exit  status,  the  second  byte  indicates  how  the  process  exited,  and  the  third  byte  is  the  exit  status. 
Otherwise,  the  final  three  bytes  in  the  header  give  the  size  of  the  packet  to  be  sent.  The 
dedicated  gareon  process,  acting  as  a  buffer  between  the  application  and  the  remote  task,  handles 
the  data  encoding. 


Sj? 


9.1.1.  Thu  Black  Box 

Although  the  fequeoce  of  coaoectioB,  •electiou,  and  output  decoding  required  by  every 
applicatioB  program  appear*  complicated,  there  it  a  fuBctioa  called  RumotuRua  which  does  it 
all: 
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RenoteRan (inFD ,  oatFD,  eadp) ; 
iat  iaFD; 

1st  oatFD; 

strsct  pipopioco  «cndp; 


where 


strsct  pipopioco  < 
char  •pp.aaao; 
char  eepp.a rga; 

> 

and  InFD  and  outFD  are  the  input  and  output  file  descriptors  that  should  be  used  for  input  to 
and  output  from  the  remote  task.  There  are  no  provisions  for  redirecting  ttderr.  Note  that 
emdp  is  a  pointer  to  struct  plpeplsce.  A  list  of  these  struets  indicates  a  sequence  of  piped 
commands,  allowing  multiple  tasks  to  be  piped  together  on  the  remote  end.  Appendix  A  shows  a 
typical  interaction  with  RemotcRun. 


4.  Operational  Considerations 
4.1.1.  Error  Handling 

Maitrc  d'  itself  does  not  have  any  provision  for  handling  errors  other  than  reporting  them  to 
the  application.  Some  common  error  situations  are: 

-  lost  connection  with  remote  host 

-  remote  garcon  prematurely  exited 

-  remote  machine  could  not  execute  process 

Whenever  possible,  an  application  should  recover  from  the  error  and  accommodate  the  user's  task 
as  quietly  and  politely  as  possible.  This  may  be  done  either  by  finding  another  host,  or  by 
processing  the  job  on  the  user's  machine.  All  of  our  applications  attempt  to  run  the  job  locally  if 
there  is  a  remote  failure. 

Because  of  redirection  facilities  and  pipes  in  UNIX,  a  difficult  situation  arises  if  the  remote 
server  is  the  recipient  of  a  local  process's  output,  and  one  of  the  errors  listed  above  occurs.  The 
process  cannot  be  restarted  in  any  way.  For  example: 

tbl  file. me  I  matfront  nroff  -me  >  file.out 

where  matfront  is  a  generic  front  end  that  attempts  to  process  its  arguments  on  a  remote 
machine  using  etdin  as  input.  Once  data  from  tW  is  passed  to  matfront,  it  cannot  be  retrieved  if 
the  remote  nroff  terminates  due  to  some  error  related  to  Maitre  d’.  Even  if  matfront  kept  a 
copy  of  tbl’i  output  (which  would  be  prohibitively  expensive),  output  already  generated  by  the 
remote  nroff  would  have  gone  to  file.out.  Any  attempt  to  restart  the  job  might  cause  duplicate 
output  to  appear  in  file.out.  Because  Maitre  (/’operates  at  the  user  level,  above  the  C-shell  and 
UNIX  kernel,  no  simple  solution  exists  for  this  problem.  Consequently,  if  an  error  occurs  after  a 
remote  job  has  started  executing,  and  the  job  is  receiving  its  input  from  a  pipe,  the  user’s  task 
terminates  with  an  error  message.  Processes  not  using  redirection  do  not  have  this  problem,  and 
can  be  restarted 
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4.1. *.  Security 

When  s  machine  service*  users  on  other  mftchioes,  various  security  problems  ftrise.  Some  of 
the  concerns  are: 

Unauthorized  Use 

Administrfttive  barriers  must  be  respected. 

Unauthorized  Access 

Use  of  spnre  cycles  should  not  allow  access  to  restricted  files. 

Unauthorized  Execution 

Not  nil  machines  should  have  to  provide  ftil  services  to  all  clients. 

Maitre  d'  solves  these  security  problems  with  client  verification,  task  verification,  non- 
privileged  users  and  logfiles. 

When  a  connection  is  requested  to  either  of  garcon ’t  two  ports,  the  originating  host  is 
checked  against  a  list  of  authorized  hostnames  contained  in  the  startup  configuration  file.  The 
request  is  ignored  if  the  host  is  not  authorized  and  the  illegal  access  attempt  is  recorded  in  a  log 
file. 

If  a  request  for  service  arrives,  garcon  guards  against  unscrupulous  applications  by  checking 
to  see  that  it  has  actually  advertised  itself  as  available.  If  not,  the  request  is  ignored.  The 
request  is  then  checked  against  a  list  of  reasonable  services  that  garcon  has  been  told  about  in  the 
configuration  file.  If  it  is  not  in  the  list,  garcon  informs  the  application  that  it  doesn't  know  about 
the  service.  It  is  then  up  to  the  application  to  either  choose  another  server  or  process  the  job 
locally. 

Associated  with  each  process  under  UNIX  is  a  user  name  governing  that  process'  access 
rights.  When  a  task  runs  under  garcon,  the  privileges  are  first  set  to  those  of  some  named  user 
given  in  the  configuration  file.  In  Berkeley’s  implementation,  all  service  tasks  run  as  nobody, 
which  is  an  actual  entry  in  the  password  file  originally  created  for  system  administration 
functions.  Nobody  has  no  password,  home  directory  or  shell  and  can  only  read  public  files.  If 
process  accounting  is  being  run  on  the  server  machines,  it  would  be  worthwhile  to  create  a 
dummy  account  used  only  by  garcon.  In  this  way,  the  standard  accounting  software  could 
determine  the  percentage  of  resources  which  are  being  used  on  behalf  of  remote  requests. 

Once  a  server  has  begun  a  remote  job,  and  if  its  load  has  risen  above  the  acceptance 
threshold,  it  is  possible  to  have  this  job's  priority  lowered  or  n-niced  during  the  period  that  the 
server  is  not  accepting  new  jobs.  Active  jobs  from  other  machines  will  then  not  impinge  upon  jobs 
coming  from  a  server  machine’s  own  users.  The  priority  is  raised  again  once  the  server’s  load  falls 
below  the  threshold. 

S.  Implementation  Notes 

All  programs  that  are  to  operate  under  the  Maitre  d'  system  require  a  front  end  (using 
RemoteRun)  on  the  client  machine  to  decode  the  returned  data.  Those  applications  that  use 
stdin,  stdout,  and  stderr  for  I/O  do  not  need  a  front  end  on  the  remote  machine,  and  may  simply 
use  mat  front  as  an  interface.  Since  Maitre  d'  supports  only  these  three  channels  of  data  transfer, 
a  few  programs  did  not  easily  integrate  into  the  system. 

1.1.  The  Compilers 
f.1.1.  C 

Creating  the  application  to  handle  C  compiles  was  relatively  trivial  due  to  the  structure  of 
the  compiler,  which  is  broken  down  into  four  distinct  parts:  pre-processing,  compilation,  assembly 
and  loading.  Pre-processing  and  loading  are  always  performed  on  the  local  machine.  The 
compilation  and  assembly,  which  comprise  almost  70%  of  the  total  cycles,  are  done  on  the  remote 
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machine. 

S.I.S.  Pascal 

A  food  example  of  a  package  that  did  not  lend  itself  well  to  running  under  Maitre  i'  was 
the  Berkeley  Pascal  interpreter  (pi).  There  were  problems  on  both  the  client  and  server  end. 

Server 

Berkeley  Pascal  (pi)  does  not  use  itdin  for  anything,  but  instead  requires  that  its  input  come 
from  a  lie  (or  several  files).  The  executable  image  produced  by  pi  goes  directly  to  a  file  and 
cannot  be  directed  elsewhere  (such  as  ttdout).  Things  are  further  complicated  by  the  fact 
that  pi  uses  ttderr  for  only  one  error  message.  All  other  error  messages  go  to  ttdout.  This 
causes  problems  for  garcon  which  expects  remote  tasks  to  communicate  back  to  the 
application  via  ttdout  k  ttderr. 

Client 

Because  of  the  way  ^include?  files  are  handled  in  pi,  it  is  semantically  legal  to  concatenate 
all  of  the  files  and  compile  them  as  a  single  stream.  Originally,  we  ran  a  very  simple  Pascal 
pre-processor  over  all  of  the  user's  source,  scanning  for  # indude  directives  and  including 
them  as  we  found  them,  sending  over  all  of  the  files  as  one  large  program.  Unfortunately, 
all  of  the  debugging  and  diagnostic  errors  produced  by  Berkeley  Pascal  have  line  numbers 
relative  to  the  beginning  of  each  source  file,  so  this  was  not  a  satisfactory  solution.  Students 
were  being  told  that  they  had  a  syntax  error  on  line  1237,  when  their  largest  file  contained 
only  250  lines. 

These  problems  were  solved  by  building  a  simple  file  transfer  protocol  on  top  of  the  three 
data  channels  already  provided  for  by  Maitre  d\  This  extra  level  required  another  layer  of  pre¬ 
processors  on  both  the  client  and  the  server  machines.  A  user  runs  Rpi  on  the  local  machine 
which  reads  the  user’s  program,  looking  for  # include  directives.  Instead  of  starting  up  pi  on  the 
server  machine,  Rpi  instead  requests  that  Spi  (server  pi)  be  executed.  Data  going  from  Rpi  to  Spi 
includes  a  header  giving  the  size  of  the  file,  the  length  of  the  file  name,  the  file  name,  and  finally 
the  file.  This  is  done  for  every  file  that  is  needed  in  the  compilation.  Spi  reconstructs  the  original 
source  code  in  a  temporary  directory  on  the  server  machine.  When  all  files  have  been  received, 
Spi  executes  pi  on  those  files  in  the  temporary  directory  and  the  compilation  is  performed.  Spi 
takes  care  of  transferring  pi’s  ttdout  to  ttderr.  If  the  compilation  completes  successfully,  Spi  reads 
the  executable  image  left  behind  by  pi  and  sends  it  to  ttdout.  This  is  picked  up  by  garcon  and 
sent  to  Rpi,  which  knows  that  anything  coming  to  ttdout  from  the  remote  machine  belongs  in  an 
executable  file  on  the  local  machine.  The  relationships  between  the  processes,  machines  and  files 
are  shown  in  figure  II. 

Maitre  d'  provides  only  a  primitive  connection  mechanism  through  which  a  task  on  a 
remote  machine  may  communicate  with  its  calling  process  on  the  local  machine.  For  tasks 
flexible  enough  to  operate  using  only  three  data  channels,  the  mechanism  is  sufficient.  The  point 
of  this  discussion  on  the  Pascal  application  ii  to  show  that  those  tasks  requiring  more  complicated 
file  access  can  be  accommodated  by  building  on  top  of  the  interface  provided.  It  may  seem 
cumbersome  and  expensive  to  transfer  all  of  a  user's  source  files  from  client  to  server  for  each 
application's  invocation.  But,  until  a  reliable  remote  file  system  becomes  widely  available,  this 
type  of  explicit  data  transfer  is  necessary.7 


*  Wbeo  the  interpreter  encounter*  i  Cat  of  the  form  "gbndude  Jflmttm "  is  the  aourn  8b,  it  reidf  the  aimed  8b 
iato  iu  input  itrtun  u  though  it  were  coded  ia-liae  by  the  u*er.|4] 

T  Even  with  i  remote  8b  (ystem,  the  diti  would  (till  need  to  be  moved  between  machine*.  The  tnnrfer  would  juft 
be  eompbtety  triaepareat. 


0.  Evaluation  ft  Performance 


6.1.  Design  Criteria 

Alooso[2|  describes  six  points  for  choosing  a  good  load-balancing  strategy:  stability, 
implementability,  cost,  autonomy,  transparency  and  tunability.  Maitre  d'  meets  all  of  these 
criteria. 

(1)  Stability.  A  server  machine  could  become  inundated  with  requests  from  clients  if  it  had 
recently  announced  its  availability.  At  instantaneous  state  information  is  very  hard  to  come 
by,  use  of  an  algorithm  that  avoids  processor  flooding  is  very  important.  Maitre  d' relies  on 
a  round-robin  selection  mechanism  of  available  hosts  to  minimiie  the  possibility  that  aoy 
one  client  might  overload  a  server.  Although  it  is  possible  for  several  clients  to  all 
simultaneously  request  the  same  server,  this  type  of  selection  synchronisation  is  highly 
unlikely. 
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(2)  Implement  ability.  Maitre  d '  runs  at  the  user  level  and  requires  do  modifications  to  the 
UNIX  kernel.  It  is  running  on  VAX  750s,  780s,  785s,  micro  VAX  IIs,  and  a  Sequent  parallel 
processor.  The  basic  package  is  about  4000  lines  of  well-commented  C  code  (approx.  50K 
bytes  object)  and  is  portable  to  any  machine  running  Berkeley  UNIX  4.2  or  4.3BSD.* 

(3)  Cost.  The  overhead  in  running  Maitre  d'  is  minimal.  There  are  three  considerations  for 
cost,  client  overhead,  network  traffic  and  server  overhead.  In  the  worst  case,  when  no  servers 
are  available,  the  user's  process  usually  requires  less  than  .5  seconds  of  real  time  to 
determine  that  the  job  must  be  performed  locally  Since  the  class  of  jobs  running  under 
Maitre  d'  are  typically  long-lived,  this  time  is  unnoticeable  (but  consistent).  Traces  show 
that  it  is  rare  for  jobs  to  require  more  than  a  few  hundred  kilobytes  of  data  to  be 
transferred  between  client  and  server  machines.  On  a  3  or  10  megabit  ethernet,  the  impact 
of  the  extra  traffic  is  minimal.  Since  servers  only  broadcast  changes  in  state  information 
and  spend  most  of  their  time  blocked  waiting  on  requests,  the  cost  of  running  the  server  is 
also  low  Using  acceptance  load  thresholds  of  5  and  2  on  our  VAX  785's  and  750  s 
respectively',  each  server  was  averaging  about  40  state  changes  a  day 

(4)  Autonomy.  The  decision  to  accept  or  reject  jobs  is  completely  up  to  the  server,  so  no 
machine  can  be  forced  to  take  on  work  from  outside  clients.  In  addition,  the  types  of  jobs  a 
server  will  accept,  and  the  client  machines  from  which  those  jobs  may  arrive  is  definable  by 
the  system  administrator  in  a  configuration  file.  The  server  machine  may  also  impose  a 
pre-emption  policy  on  remote  jobs,  always  giving  priority  to  its  own  users. 

(5)  Transparency.  The  fact  that  a  process  is  being  executed  on  a  remote  machine  can  be  made 
completely  transparent  to  the  user.  Users  need  never  know  that  their  tasks  are  being 
performed  elsewhere. 

(6)  Tunability.  System  parameters  can  be  set  and  reset  through  a  configuration  file  and  a 
dynamic  control  program  (mdc).  Thus,  Maitre  d'  can  be  run  in  various  environments 
without  the  need  for  recompilation. 


8.2.  Performance 

Several  factors  contribute  to  the  success  of  Maitre  d'.  First,  the  decision  to  export  only 
high-cost,  CPU  intensive  jobs  allows  a  machine,  regardless  of  how  busy  it  might  be,  to  provide 
swift  response  time  for  those  jobs.  As  the  less  expensive,  non-ported  jobs  are  no  longer  competing 
with  the  more  costly  ones  for  resources,  they  too  enjoy  an  improvement  in  response  time.  Table  I 
shows  comparative  statistics  for  April  1984  and  April  1985  taken  on  a  VAX  11/780  (ucbcory).  In 
1984,  the  machine  ran  without  Maitre  d'.  In  April  1985,  the  only  jobs  being  offloaded  were 
Pascal  and  C  compiles.  About  3000  compiles  per  week  were  being  exported  to  two  VAX  750's 
operating  as  dedicated  remote  servers.  Given  are  the  UNIX  five  minute  load  averages;  the  times 
to  start  up  the  editor  on  a  trivial  file,  and  the  times  (in  seconds)  to  compile  and  execute  locally 
the  following  short  CPU  intensive  program: 


*  There  were  tome  problem*  initially  with  the  Sequent,  but  it  turned  out  to  be  *  bug  in  their  4.2  releaae  ud  Dot  is 
Mtitrt  4 

*  Experience  hu  shown  these  loads  to  be  comfortable  operating  points.  Below  these  values,  the  machines  tend 
toward  idleness  Above  them,  response  time  becomes  intolerable. 
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■aiaO 

< 


> 


lat  l.J.a; 

fer(i«0  ;  K1000  ;  i**> 

for  (J«0  ;  J<1000  ;  j**) 

wl; 


The  demands  on  the  machine  from  instructional  coursework  were  identical  for  the  two 
months  being  compared.  The  increase  in  performance  can  be  seen  across  the  board  in  all  the 
statistics,  with  an  average  improvement  factor  of  over  2.  The  increase  in  perceived  machine 
performance  is  even  more  dramatic  considering  that  the  VAX  780  was  running  with  16  megabytes 
of  memory  in  1084,  but  only  half  that  during  the  1085  sampling  period.  There  were  no  other 
configuration  changes.  The  decrease  in  variance  for  all  the  figures  demonstrates  that  a  balanced 
system  has  the  added  feature  of  offering  more  predictable  response  times. 


Table  I 

VAX  11/780  Performance  Comparisons 


uebeory 

MESSEMM 

Improvement 

loBSSHfSI 

loEBESBI 

Factor 

#  samples 

2140 

2134 

• 

mean  users 

20.01 

. 

median  users 

20 

• 

variance  users 

97 

100 

- 

mean  load 

6.12 

2.52 

2.42 

median  load 

4.6 

1.8 

• 

variance  load 

23.95 

6.07 

- 

mean  editor  (secs) 

1.46 

.82 

178 

median  editor 

1 

1 

- 

variance  editor 

1.87 

.362 

- 

mean  compile  (secs) 

11.90 

7.04 

1.69 

median  compile 

8 

6 

- 

variance  compile 

94.6 

20.42 

- 

mean  execution  (secs) 

63 

26.7 

2.35 

median  execution 

30 

14 

- 

variance  execution 

5564 

1142 

- 

The  presence  of  lightly  loaded  machines,  and  Maitre  d's  ability  to  locate  them  also 
contributes  to  its  success.  Across  even  as  few  as  six  machines,  the  likelihood  that  one  of  them  will 
be  idle  at  any  given  time  is  fairly  high.  This  is  partially  because  of  the  type  of  workload  imposed 
by  instructional  computing,  i.e.,  a  machine  is  very  busy  shortly  before  an  assignment  is  due,  and 
much  less  so  at  other  times.  This  can  be  seen  in  appendix  F,  where  the  five  minute  load  average 
( Maitre  d\  key  value)  is  given  for  six  machines  over  a  one  month  period.  These  figures  were 
taken  in  March  1984  on  machines  not  running  Maitre  d\  After  Maitre  d’  was  in  place,  our 
busiest  machine  (a  VAX  11/785)  performed  66709  compiles,  38524  of  them  remotely,  in  just  96 
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days.  The  less  powerful  ll/7S0's  would  average  between  three  sad  Ive  thousand  compiles  per 
machine  per  week.  An  overloaded  machine  looked  to  Bad  an  idle  remote  processor  without 
success  on  only  08  occasions.  This  indicates  that,  for  the  most  part,  at  least  one  lightly  loaded 
machine  could  always  be  found  among  the  six  possible  servers. 

One  metric  used  to  gauge  the  relative  utilization  of  machines  over  a  long  period  of  time  is 
the  mean  load  average.  This  is  simply  the  average  value  of  a  machine's  load  average  when 
sampled  at  regular  intervals.  From  this  value  we  also  found: 

(a)  that  those  machines  whose  mean  load  average  before  Maitre  d'  was  less  than  garcon's 

acceptance  threshold  value  tended  to  have  their  mean  increase  after  load  sharing  was  in 

place,  and 

(b)  that  those  machines  whose  mean  load  average  was  above  maitrd' s  sendoff  threshold  saw  a 

decrease  in  their  mean  load  average,  as  well  as  a  marked  decrease  in  the  variance  of  the  load 

average. 

This  simply  means  that  those  machines  most  in  need  of  assistance  will  benefit  the  most  from 
load  sharing,  and  machines  which  were  idle  will  find  themselves  more  busy.  This  is  exactly  what 
should  be  happening  and  is  not  surprising. 


7.  Conclusions 

We  have  implemented  an  effective  load-balancing  system  and  demonstrated  its  utility. 
Further  applications  can  be  easily  introduced  to  operate  within  the  package's  environment. 
Performance  metrics  indicate  that  this  system  is  highly  successful  in  creating  a  more  responsive 
and  pleasant  working  environment  for  users. 
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Appendix  A 


Using  RemoteRun 


flaclade  'dsfi.k*  /•  for  Muii[  of  ReaoteRaa'a  rotara  r alias  a/ 


aala  (»rgc,  argr) 
irt  ttc; 
o’  it(t; 

< 

iat  r; 

let  rarbosa  ■  0; 
lat  aldray  ■  0; 

■tract  plpeplece  PlpeLlst[10] ; 


MakaPlpaLlat(argT,  PlpaLlst);  /a  antoaagically  con Tarts  argr  a/ 

/a  to  pipe  Hat  a/ 


r  a  RaaotaRaa(0,  1,  PlpaLlst);  /a  laFD  *=  stdla  a/ 

/•  oatFD  n  stdont  a/ 

If  (r)  { 

If  (r  **  -1)  { 

If  (rarboaa) 

fprlatf (atdarr,*Xa:  Error  la  coaaectlag  to  raaota  aachlaa\a*.argT[0]); 
do. local (argr); 

>  also  ll(r«  -X. ABORT)  { 

fprlatf (stderr,*|s:  Raaota  aerrer  killed  by  signal\a.a,argr[0]); 

aldray**; 

rarboaa**; 

do. local (argr) ; 

)  else  If  (r  »»  -X.EXEC)  { 

fprlatf (atdarr.'la:  Raaota  failed  to  exec. \a *, argr [0]); 

rarboaa** ; 

do.  local  (argr) ; 

>  also  if  (r  ■«  -X.I0F1D)  { 

If  (rarboaa) 

fprlatf (stderr.'fa:  Raaota  d‘aaa‘t  know  aboot  this  coaaand.\na,argr[0]); 
do. local (argr); 

}  else  If  (r  **  X.LOCAL)  { 
do. local (argr) ; 

)  else  If  (r  «  X.RADEfT)  < 
aldray** ; 
rarboaa**; 
do.  local  (argr) ; 

>  else  If  (r  «  -X.MIDIAT)  { 

fprlatf (atdarr,*\a\007\007Ia:  Lost  connection  with  raaota  hoat\na,argT[0]); 
aldray**; 
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verbose**; 
do.  local  (argv) ; 

) 

) 

exit(r) ; 

> 


do. local (argv) 

char  eeargv; 

< 

/• 

•  caa  only  continue  if  lnpat  coning  fron  file  and  not 

•  a  pipe  and  tho  procenn  had  not  jot  started. 

•/ 

if  (  aldvay  U  I (isattj(O)  U  inatttj(l))  )  { 

fprintf (stderr, 'Broken  pipe  (renote)\n() ; 
exlt(-l);  /•  nser  ontta  luck  •/ 

> 

if  (verbose) 

fprintf (stderr . 'Processing  locallj\n*) ; 

execvp(*argv,  argv); 

/«  IOTREACHED  */ 
perror ( 'natf root : • ) ; 

fprintf (stderr, *Conld  not  exeente  |s\n*.  eargv); 
exit(-l); 

> 
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Semple  Configuration  File 


•  DCBSIM 

•  Maitrd  Lead 


Balancing  Configuration  File 

Inatnctloin  arallabla  to  oitnldo  world 

Keonnaad  nano  coaaand  path  proc  ald> 
Poaalblo  Client 

C<acbxytny> 

People  Serrera  for  na 

S<acbxjxzj>  ...  no  entrlea  aeana  all 

Entry  both  a  client  and  a  earner 
B<acbzyzzy> 

Carcon  configuration  option 
C<option>  <Talie> 

Clearing  Bonae  Machine 

n<icbxyzzy> 


•  Theae  nachlnea  are  allowed  in: 

Cncbear 

Cacbeaat 

Cicbdall 

Cacbarpa 

•  Theae  eachlaea  ahonld  be  willing  to  take  joba  fron  na: 
Sncbfraany 

Sncbzooej 

Sicbain 

Cucbbaddy 

0  Thin  gay  takea  and  receives: 

•acbernie 


•  Sequent  fron  (localhoat  «*  onraelwee) 

alocalhoat 


• 

00 

0 

0 

0 


00000000000000000000001 
Comuada  Ve  Can  Inn 


00 


0  Anything  here  can  be  ran  by  people  on  the  outalde 
0  Flrat  colann  *  nano  by  reqaeat 
0  Second  colann  ■  path  of  aaaoclated  prograa 
0  Third  colann  *  near  nano  for  taak 
0 


0 

0 

Icah  /bla/cah  root 

weald  be  a  bad  entry 

0 

Xccoa 

/llb/ccon 

nobody 

Inroff 

/aar/bln/nroff 

nobody 

Icat 

/bln /cat 

nobody 

Idate 

/bln/date 

nobody 

Xecho 

/bla/echo 

nobody 

Iwhoani 

/aar/acb/whoanl 

nobody 

Appendix  B 


Ihostaaae 

/bia/hostsaas 

aobody 

ISpi 

/wsr/local/Spi 

aobody 

•  pascal  frost  sad 

IScc 

/wsr/local/Scc 

aobody 

•  C  frost  sad 

Itroff 

/wsr/bia/troff 

aobodj 

Itroff.p 

/asr/local/troff.p 

aobody 

•  ditroff 

Irwho 

/usr/web/rwho 

aobody 

Ispice 

• 

/cad/bia/spice 

aobody 

•  circuit  siaalatioa 

• 

•  Server  reitiae  optioie .... 

• 

i  Load  threshold  above  vhieh  Jobs  are  aot  accepted 
Cload  S.O 

•  luber  of  non -idle  users  above  which  Jobs  are  sot  accepted 

Gssers  20 

•  Maxinnn  b saber  of  clieats  allowed  is 

Gclieats  15 

•  Vhea  the  load  exceeds  threshold,  raise  priority  (reaice)  of 

•  active  Jobs  to  arguaeat.  Is  saix,  this  is  aaalogoas  to  proenptios. 

Csice  4 
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NAME 

Rpi  -  local  front  end  for  using  pi  (pascal)  with  maitrd 
Spi  -  server  front  end  for  using  pi  (pascal)  with  garcon 

SYNOPSIS 

Rpl  [  +gvd  +o  [file]  -[pi  options]  ] 

Spl  [  +fv  -[pi  options]  ] 

DESCRIPTION 

Rpi  is  a  "user-  friendly”  interface  for  using  pi  with  the  maitrd/garcon  load  balancing  software.  It 
finds  an  available  machine  for  processing,  transfers  files  needed  by  pi,  and  brings  back  any  output 
generated  by  the  remote  pi.  Before  beginning  the  compile,  Rpi  copies  any  existing  obj  file  to 
obj.bak. 

In  its  simplest  form,  it  is  invoked  just  as  pi  is: 

Rpi  prog.p 

Since  pi  requires  as  input  a  file  ending  in  .p,  Rpi  does  not  call  up  pi  immediately,  but  instead 
requests  to  run  Spi  on  the  server  machine.  Rpi  and  Spi  transfer  needed  source  files  and  new 
object  files  to  one  another.  Spi  creates  a  temporary  work  directory  on  the  server  end.  It  is  here 
where  the  pi  is  actually  performed.  Pi  error  messages  are  routed  back  to  the  user  through  stderr. 
If  the  compilation  was  successful  (an  obj  file  exists  in  the  remote  work  directory),  the  obj  file  is 
moved  back  to  the  current  working  directory.  Only  a  single  status  message  from  the  remote 
machine  gives  any  indication  that  the  program  is  not  being  compiled  locally. 

Spi  can  not  be  used  directly,  as  it  expects  data  to  be  headed  with  filestat  information.  In  this 
manner,  multiple  files  can  be  passed  though  a  single  pipe.  / 

Pascal  include  files  cause  numerous  headaches  with  remote  processing.  The  source  files  must  all 
be  scanned  for  #include  directives.  As  all  work  is  done  in  a  single  directory  on  the  remote  end, 
including  files  from  a  directory  other  than  the  current  one  can  cause  ’minor”  problems.  File 
pathnames  are  modified  to  replace  all  occurrences  of  ’/'  with  '\'  to  maintain  unique  names  in  the 
flat  name  space  on  the  remote  machine.  When  the  compilation  completes,  the  executable  s  symbol 
table  is  examined  for  all  references  to  ’\'.  These  are  changed  back  to  '/'.  Munging  the  filename 
in  this  way  has  two  consequences.  A  filename  is  not  allowed  to  include  the  special  character  '\', 
and  all  error  messages  generated  by  the  compiler  referencing  the  munged  file  will  have  '\'s 
replacing  '/’*•  So,  an  error  in  file  /a/b/c/d/file.i  will  be  reported  as  an  error  in  \a\b\c\d\file.i. 

OPTIONS 

+(  When  the  server  has  completed  compiling,  it  will  normally  clean  up  whatever  workspace 
it  required.  If  the  +g  option  is  used,  garbage  generated  by  the  compilation  will  be  left 
behind,  and  the  workspace  path  will  be  given  upon  program  termination.  This  is  only 
really  useful  for  debugging. 

+o  Normally,  Rpi  output  will  go  into  obj.  This  option  allows  you  to  specify  the  output  file. 

+v  This  puts  Rpi  into  verbose  mode.  It  can  sometimes  be  fun  to  watch  if  you  like  this  sort 

of  stuff. 

+d  This  sets  a  debugging  flag.  It  is  not  used  for  much,  +v  is  better. 

-[pi  options] 

Any  twitches  preceeded  by  a  -  will  be  passed  through  untouched  to  the  pi  program. 


ERRORS 

If,  for  any  detectable  reason,  the  remote  end  can  not  perform  the  compilation  (host  dies,  failed 
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SEE  ALSO 

maitrd(l),  garcon(I),  Rpi(l),  Rcc(l) 

DIAGNOSTICS 

TAmbiguous  command  abbreviation  matches  more  than  one  command 

’Invalid  command  no  match  was  found 

^Privileged  command  command  can  be  executed  by  root  only 

BUGS 

The  'machine'  command  is  probably  silly. 
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NAME 

mdc  -  maitrd  control  program 
SYNOPSIS 

/uar/Iocal/mde  [  command  [  argument  ...  ]  j 
DESCRIPTION 

Mdc  is  used  by  tbe  system  administrator  to  control  the  operation  of  the  maitrd  load  balancing 
software.  For  any  machine  running  tbe  maitrd  client  daemon,  mdc  may  be  used  to: 

•  set  the  dynamic  load  threshold  at  which  jobs  are  exported, 

•  force  the  daemon  to  reread  the  configuration  file,  restarting  all  active  connections,  and 
attempting  to  reestablish  dormant  ones, 

•  display  the  current  status  of  the  maitrd  daemon, 

•  kill  the  daemon  without  restarting  it. 

Without  any  arguments,  mdc  will  prompt  for  commands  from  the  standard  input.  If  arguments 
are  supplied,  mdc  interprets  the  first  argument  as  a  command  and  the  remaining  arguments  as 
parameters  to  the  command.  The  standard  input  may  be  redirected  causing  mdc  to  read 
commands  from  file.  Commands  to  mdc  may  be  sent  to  any  machine  running  the  maitrd  software. 
If  no  machine  is  given  explicitly  on  the  command  line,  mdc  directs  the  command  to  the  last 
referenced  machine.  If  no  machine  has  yet  been  referenced,  then  tbe  local  host  is  assumed.  Any 
number  of  hosts  may  be  given  on  a  command  line.  Mdc  will  send  the  command  to  each  host. 
Commands  may  be  abbreviated;  the  following  is  the  list  of  recognized  commands. 

I  [  command  ...  ] 

help  J  command  ...  ] 

Print  a  short  description  of  each  command  specified  in  the  argument  list,  or,  if  no 
arguments  are  given,  a  list  of  tbe  recognized  commands.  * 

kill  {  (host)*  } 

Terminate  the  active  maitrd  daemon  at  tbe  host  (or  hosts)  immediately.  This  command 
is  restricted  to  the  superuser. 

load  #  {  (host)*  } 

Set  the  load  threshold  to  the  second  argument  at  each  of  tbe  indicated  (or  implied)  hosts. 
This  command  is  restricted  to  tbe  superuser.  » 

exit 

quit 

Exit  from  mdc. 
restart  {  (host)*  } 

This  will  cause  the  maitrd  daemon  at  tbe  indicated  hosts  to  reread  the  configuration  file, 
close  all  existing  connections  and  attempt  to  reestablish  connections  with  each  host  given 
in  the  configuration  file.  All  in-core  statistics  are  zeroed.  Jobs  in  the  middle  of 
processing  are  unaffected  and  will  continue  normally. 

status  {  (host)*  }  . 

This  displays  the  status  of  the  maitrd  daemon  at  each  indicated  host.  This  is  an 
unrestricted  command. 

machine  {  host  } 

This  sets  the  default  host  to  its  argument.  With  no  arguments,  it  returns  tbe  current 
default  host. 

FILES 

/usr/local/maitrd.conf  maitrd  configuration  file 
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exec,  etc...),  the  compilation  will  be  performed  locally. 

BEE  ALSO 

maitrd(l),  garcoa(l),  matfront(l),  mdc(l),  tocket(2),  pi(l) 

BUGS 

The  translation  between  ’/'  *V  **  annoying. 

The  (g)arbage  twitch  it  really  naelesa  unless  there  is  the  capability  to  return  the  entire  temporary 
directory  back  to  the  local  machine. 

Error  messages  from  the  compiler  are  tent  to  standard  error.  Any  output  from  Rpi  must  be 
redirected  with  "l&"  or  . 
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NAME 

Rcc  -  Remote  C  Compiler 
See  -  Server  C  Compiler 

SYNOPSIS 

Rcc  (  option  ]  ...  file  ... 

DESCRIPTION 

Rcc  is  the  UNIX  remote  C  compiler.  Rcc  accepts  several  types  of  arguments: 

Arguments  whose  names  end  with  ‘.c'  are  taken  to  be  C  source  programs;  they  are  compiled,  and 
each  object  program  is  left  on  the  file  whose  name  is  that  of  the  source  with  ‘.o'  substituted  for 
'.c‘.  The  ‘.o'  file  is  normally  deleted,  however,  if  a  single  C  program  is  compiled  and  loaded  all  at 
one  go. 

In  the  same  way,  arguments  whose  names  end  with  '.s’  are  taken  to  be  assembly  source  programs 
and  are  assembled,  producing  a  ‘.o'  file. 

Rcc  (capital  R)  is  not  the  same  as  rcc  (lower  case  r).  The  latter  fires  up  shells  on  remote  machines 
and  requires  that  the  user  have  an  account  there.  Rcc  is  written  to  work  with  the  Maitrd  remote 
server  software  and  requires  only  that  there  be  machines  willing  to  accept  job  requests.  When 
invoked,  Rcc  checks  to  see  if  the  load  is  low  enough  to  run  the  compile  locally.  If  so,  Rcc  acts 
just  like  cc.  If  the  local  load  is  too  high,  Rcc  will  attempt  to  locate  a  foreign  machine  that  is  not 
too  busy  and  execute  much  of  the  compilation  at  the  remote  host.  If  such  a  machine  can  not  be 
found,  or  if  the  connection  becomes  'fiakey'  or  lost,  Rcc  will  force  the  remote  compilation  to  be 
done  locally.  See  is  used  to  perform  the  remote  compilation  and  is  called  automatically  from  Rcc. 
See  should  not  be  run  stand-alone. 

C  preprocessing  (epp)  and  loading  (Id)  are  always  done  locally.  Only  compilation  (ccom)  and 
assembly  (as)  are  done  on  the  remote  machine.  If  Rcc  is  invoked  with  the  -S  option,  assembly  on 
the  remote  end  will  be  bypassed.  If  the  input  file  is  only  an  assembly  source  program  (ends  in 
‘.s’),  Rcc  will  not  even  attempt  to  assemble  it  remotely.  It  will  be  assembled  locally. 


The  following  options  are  interpreted  by  Rcc  just  like  in  cc.  See  ld(l)  for  load-time  options 

Suppress  the  loading  phase  of  the  compilation,  and  force  an  object  file  to  be  produced 
even  if  only  one  program  is  compiled.  v 

—g  Have  the  compiler  produce  additional  symbol  table  information  for  dbz{  1).  Also  pass  the 
-Ig  flag  to  ld(  1). 

-go  Have  the  compiler  produce  additional  symbol  table  information  for  Ibe  obsolete  debugger 
«dh(l).  Also  pass  the  -If  Sag  to  ld(  1). 

-w  Suppress  warning  diagnostics. 

-p  Arrange  for  the  compiler  to  produce  code  which  counts  the  number  of  times  each  routine 
is  called.  If  loading  takes  place,  replace  the  standard  startup  routine  by  one  which 
automatically  calls  moHitor(3)  at  the  start  and  arranges  to  write  out  a  men. out  file  at 
normal  termination  of  execution  of  the  object  program.  An  execution  profile  can  then  be 
generated  by  use  of  prof(  1 ). 

-pg  Causes  the  compiler  to  produce  counting  code  in  the  manner  of  -p,  but  invokes  a  run¬ 
time  recording  mechanism  that  keeps  more  extensive  statistics  and  produces  a  gmon.out 
file  at  normal  termination.  Also,  a  profiling  library  is  searched,  in  lieu  of  the  standard  C 
library.  An  execution  profile  can  then  be  generated  by  use  of  gprof(  1) 

-O  Invoke  an  object-code  improver. 


4th  Berkeley  distribution 


20  February  1984 


1 


f. 

r. 

K 


RCC(l) 


UNIX  Programmer'*  Manual 


RCC(I) 


-R  Passed  on  to  as,  making  initialized  variable*  (hared  and  read-only. 

-a  Compile  the  named  C  programs,  and  leave  the  assembler-language  output  on 

corresponding  flies  suffixed  's'. 

-E  Run  only  the  macro  preprocessor  on  the  named  C  programs,  and  send  the  result  to  the 
standard  output. 

— C  prevent  the  macro  preprocessor  from  eliding  comments. 

— o  output 

Name  the  Anal  output  file  output.  If  this  option  is  used  the  file  'a.out'  will  be  left 
undisturbed. 

-Dnamt^def 

-Dnamt 

Define  the  name  to  the  preprocessor,  as  if  by  *#deflne’.  If  no  definition  is  given,  the 
name  is  defined  as  *1*. 


-Uname 
-Idir 

—Bttring 

I 

c 

-t(pOXZj 

-d 
-v 


Remove  any  initial  definition  of  name. 

'#include'  files  whose  names  do  not  begin  with  '/'  are  always  sought  first  in  the  directory 
of  the  file  argument,  then  in  directories  named  in  -I  options,  then  in  directories  on  a 
standard  list. 

7 

Find  substitute  compiler  passes  in  the  files  named  string  with  the  suffixes  cpp,  ccom  and 
c2.  If  string  is  empty,  use  a  standard  backup  version. 

I 

Find  only  the  designated  compiler  passes  in  th»  files  whose  names  are  constructed  by  a 
-B  option.  In  the  absence  of  a  -B  option,  the  string  is  taken  to  be  ‘/usr/c/’. 

Run  in  debugging  mode  showing  the  names  of  intermediate  files  as  they  are  created. 

Run  in  verbose  mode,  with  the  remote  end  indicating  its  actions  as  it  goes  along. 


Other  arguments  are  taken  to  be  either  loader  option  arguments,  or  C-compatible  object 
programs,  typically  produced  by  an  earlier  cc  run,  or  perhaps  libraries  of  C-compatible  routine^. 
These  programs,  together  with  the  results  of  any  compilations  specified,  are  loaded  (in  the  order 
given)  to  produce  an  executable  program  with  name  a.out. 


FILES 


file.c  input  file 

flle.o  object  file 

a.out  loaded  output 

/tmp/ctm?  temporary 

/lib/cpp  preprocessor 

/lib/ccom  compiler 

/usr/c/occom  backup  compiler 

/usr/c/ocpp  backup  preprocessor 

/lib/c2  optional  optimizer 

/lib/ertO.o  runtime  startoff 

/lib/mertO  o  startoff  for  profiling 

/usr/lib/gcrtO.ostartoff  for  gprof-profiling 

/lib/libc.a  standard  library,  see  intro(3) 

/usr/lib/libc.p  (profiling  library,  see  tn(ro(3) 

/usr/include  standard  directory  for  ‘#include’  files 

mon.out  file  produced  for  analysis  by  prof(  1) 
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gmoo.out  file  produced  for  analysis  by  gprof(  1) 

SEE  ALSO 

B.  W.  Kernighan  and  D.  M.  Ritchie,  The  C  Programming  Language,  Prentice-Hall,  1978 
B.  W.  Kernighan,  Programming  in  C — a  tutorial 
D.  M.  Ritchie,  C  Reference  Manual 

monitor(3),  prof(l),  gprof(l),  adb(l),  ld(l),  dbx(l),  as(l),  maitrd(l),  garcon(l) 

DIAGNOSTICS 

The  diagnostics  produced  by  C  itself  are  intended  to  be  self-explanatory.  Occasional  messages 
may  be  produced  by  the  assembler  or  loader.  Any  problems  occurring  due  to  the  migration  of 
program  source  should  result  in  the  compilation  being  performed  locally.  Unless  the  -v  option  ts 
set,  this  transition  should  be  completely  transparent. 

BUGS 

The  compiler  currently  ignores  advice  to  put  char,  unsigned  char,  short  or  unsigned  short 
variables  in  registers.  It  previously  produced  poor,  and  in  some  cases  incorrect,  code  for  such 
declarations. 

Each  file  on  the  argument  line  is  compiled  separately,  so  the  final  program  may  have  been 
compiled  on  many  different  machines.  This  can  be  considered  as  either  a  bug  or  a  feature. 
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